Post - Inference Methods for Scalable Probabilistic Modeling

Post-Inference Methods for Scalable Probabilistic Modeling by Willie Neiswanger This thesis focuses on post-inference methods, which are procedures that can be applied after the completion of standard inference algorithms to allow for increased efficiency, accuracy, or parallelism when learning probabilistic models of big data sets. These methods also aim to allow for efficient computation given distributed or streaming data, and given models that incorporate complex prior information. A few examples include: ˆ Embarrassingly parallel inference. Large data sets are often distributed over a collection of machines. We first compute an inference result (e.g. with Markov chain Monte Carlo or variational inference) on each machine, in parallel, without communication between machines. Afterwards, we combine the results to yield an inference result for the full data set. ˆ Prior swapping. Certain model priors limit the number of applicable inference algorithms, or increase their computational cost. We first choose any “convenient prior” (e.g. a conjugate prior, or a prior that allows for computationally cheap inference), and compute an inference result. Afterwards, we use this result to efficiently perform inference with other, more sophisticated priors or regularizers. ˆ Local posterior revisions. After inferring an approximate posterior density via some inference algorithm, we may want to reduce the error of this approximation in certain regions of the parameter space to increase the accuracy of posterior expectations or performance of post-inference methods. We develop efficient methods for local revisions of posterior approximations. We also describe the benefits of combining the above methods, present methodology for applying the embarrassingly parallel procedures when the number of machines is dynamic or unknown at inference time, develop randomized algorithms for efficient application of post-inference methods in distributed environments, show ways to optimize these methods by incorporating test-functions of interest, and demonstrate how these methods can be implemented in probabilistic programming frameworks for automatic deployment.